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IN THE CLAIMS : 

Please amend Claims 1, 20, and 39 as follows. 

1. (Currently Amended) A simulator apparatus with which an operator plays a 
simulation with virtual object(s) in mixed reality space including a virtual space and real space, 
said simulator comprising: 

a viewpoint detection unit adapted to detect the location/posture of a viewpoint of 

the operator; 

an inputting unit adapted to input a real space image corresponding to the 
location/posture of a viewpoint of the operator; 

a geometric information acquisition unit adapted to recognize geometric 
information of real object(s); 

a rule memory adapted to store rules for controlling the action of the virtual 

object(s); 

a computation unit adapted to determine the next action of the virtual object(s) by 
referring to said rule memory based on a relation among the location/posture of a viewpoint of 
the operator, location(s) of the virtual object(s) and the geometric information of the real 
object(s); and 

a presentation unit adapted to generate at least one image of the virtual object(s) 
on the basis of the location/posture of the virtual object(s) after the determined action and the 
location/posture of the viewpoint position of the operator, and to represent the mixed reality 
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space to the operator by superimposing the image(s) of the virtual object(s) on the operator's 
view of the real space, 

wherein said rules control the action of the virtual object(s) on the basis of an 
objective and a relative location relationship between the virtual object(s) and the real object(s) 
that is represented by the geometric information . 

2. (Previously Presented) The apparatus according to claim 1 , wherein said 

presentation unit further comprises: 

an image-capturing unit adapted to capture real space images of said operator's 
view of real space images of said player's view of the real space; 

an image generation unit adapted to generate mixed reality images representing 
the mixed reality space by superimposing or overlaying said image(s) of the virtual object(s) on 
said real space images; and 

a video see-through type display that the operator wears wherein said mixed 
reality images are displayed. 

3. (Previously Presented) The apparatus according to claim 1, wherein said 
presentation unit further comprises an optical see-through type display that the operator wears 
wherein said virtual object image(s) are displayed. 



4. (Previously Presented) The apparatus according to claim 1, further comprising, 
a status detector that detects a status of the operator; 
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wherein said computation unit determines a next action of the virtual object in 
accordance with the rule stored in said rule memory and in correspondence with the 
location/posture of the real object and/or the status of the operator, and computes a 
location/posture of the virtual object after the determined action. 



Claims 5-7 (Canceled) 



8. (Previously Presented) The apparatus according to claim 1, wherein the real 
object(s) include other operators who operate said simulator apparatus, and the another operators 
share a single mixed reality space with the operator. 

9. (Previously Presented) The apparatus according to claim 1, wherein the real 
object is an object which is fixed in position, and 

said geometric information acquisition unit comprises: 

a predetermined memory for pre-storing location information and shape 

information of the real object; and 

a reading unit that reads out the location information and shape information of the 

real object from said predetermined memory as needed. 



10. (Previously Presented) The apparatus according to claim 1, wherein the real 
object is an object which is movable but does not deform, and 

said geometric information acquisition unit comprises: 

-4- 



a predetermined memory for pre-storing shape information of the real object; 
a location/posture sensor for detecting a location/posture of the real object; and 
a setting unit that sets a region the real object is expected to occupy in the 
mixed real space in accordance with the detected location/posture of the real object. 

Claims 11-17 (Canceled) 

18. (Previously Presented) The apparatus according to claim 1, wherein said 
viewpoint detection unit detects a location/posture of the head of the operator, and 

said apparatus further comprises a detector that detects a location/posture of a 
hand of the operator; and 

a recognition unit adapted to recognize a relative location of the hand of the 
operator with respect to the head as a command on the basis of an output from said detector. 

19. (Previously Presented) The apparatus according to claim 1, wherein said 
presentation unit comprises: 

an alignment unit that aligns the location/posture of the real object to the 
location/posture of the virtual object after movement; 

a generation unit that generates an image of the virtual object after alignment in 
correspondence with an occlusion relationship; and 

a head-mounted display device. 
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20. (Currently Amended) An image processing method for a simulator apparatus 
with which an operator plays a game with virtual object(s) in a mixed reality space comprising: 
a viewpoint detection step for detecting the location/posture of a viewpoint of the 

operator; 

an inputting step for inputting a real space image corresponding to the 
location/posture of a viewpoint of the operator; 

a geometric information acquisition step for acquiring geometric information of 

real object(s); 

a computation step for determining the next action of the virtual object(s) by 
referring to the rule memory based on a relation among the location/posture of a viewpoint of the 
operator, location(s) of the virtual object(s) and the geometric information of the real object(s); 
and 

a presentation step for generating at least one image of the virtual object(s) on the 
basis of the location/posture of the virtual object(s) after the determined action and the 
location/posture of the viewpoint position of the operator, and for representing the mixed reality 
space to the operator by superimposing the image(s) of virtual object(s) on the operator's view of 
the real space a 

wherein said rules control the action of the virtual objectfs) on the basis of an 
objective and a relative location relationship between the virtual obiectfs) and the real obiect(s) 
that is represented by the geometric information . 



21. (Previously Presented) The method according to claim 20, wherein the 
operator wears a video see-through type display, and said presentation step further comprising, 
image-capturing step for capturing real space images of said operator's view of 

the real space; 

image generation step for generating mixed reality images representing of the 
mixed reality space by superimposing or overlaying said image(s) of virtual object(s) on said real 
space images and for displaying said mixed reality images on the display. 



22. (Previously Presented) The method according claim 20, wherein the operator 
wears an optical see-through type display and said presentation step representing the mixed 
reality space to the operator by displaying the image(s) of virtual object(s) on the display. 

23. (Previously Presented) The method according to claim 20, further 

comprising, 

status detecting step for detecting status of the operator; 

wherein said computation step determines a next action of the virtual object in 
accordance with the rule stored in said rule memory and in correspondence with the 
location/posture of the real object and/or the status of the operator, and computing a 
location/posture of the virtual object after the determined action. 



Claims 24-26 (Canceled) 



27. (Previously Presented) The method according to claim 20, wherein the real 
object(s) include other operators who operate the simulator apparatus, and wherein the other 
operators share a single mixed reality space with the operator. 



28. (Previously Presented) The method according to claim 20, wherein the real 
object is an object which is fixed in position, and 

the geometric information acquisition step includes the steps of: 
pre-storing location information and shape information of the real object in a 

predetermined memory; and 

reading out the location information and shape information of the real object from 

the predetermined memory as needed. 

29. (Original) The method according to claim 20, wherein the real object is an 
object which is movable but does not deform, and the geometric information acquisition step 
includes the steps of: 

pre-storing shape information of the real object in a predetermined memory; 
detecting a location/posture of the real object by a location/posture sensor; and 
setting a region the real object is expected to occupy in the mixed real space in 
accordance with the detected location/posture of the real object. 

Claims 30-36 (Canceled) 



37. (Previously Presented) The method according to claim 20, wherein the 
viewpoint detection step includes the step of detecting a location/posture of the head of the 
operator , and 

said method further comprises the detection step of detecting a location/posture of 
a hand of the operator; and 

the step of recognizing a relative location of the hand of the operator with respect 
to the head as a command on the basis of an output in the detection step. 

38. (Original) The method according to claim 20, wherein the presentation step 
includes the steps of: 

aligning the location/posture of the real object to the location/posture of the virtual 
object after movement; and 

generating an image of the virtual object after alignment in correspondence with 
an occlusion relationship. 

39. (Currently Amended) A storage medium which stores a program of an image 
processing method for a simulator apparatus with which an operator plays a simulation with 
virtual object(s) in a mixed reality space including a virtual space and real space comprising: 

a viewpoint detection program step for detecting the location/posture of a 
viewpoint of the operator; 

an inputting program step for inputting a real space image corresponding to the 
location/posture of a viewpoint of the operator; 
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a geometric information acquisition program step for acquiring geometric 
information of real object(s); 

a computation program step for determining the next action of the virtual object(s) 
by referring to rules for controlling the action of the virtual object(s) based on a relation among 
the location/posture of a viewpoint of the operator, location(s) of the virtual object(s) and the 
geometric information of the real object(s); and 

a presentation program step for generating at least one image of the virtual object 
on the basis of the location/posture of the virtual object(s) after the action and the 
location/posture of the viewpoint position of the operator, and for representing the mixed reality 
space to the operator by superimposing the image(s) of the virtual object(s) on the operator's 
view of the real space, 

wherein said rules control the action of the virtual object(s) on the basis of an 
objective and a relative location relationship between the virtual object(s) and the real object(s) 
that is represented by the geometric information . 
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